System And Method For Viewing A Subject

ABSTRACT

A viewing system or imaging system is disclosed that includes optical pieces for viewing a subject. The viewing system may include features that allow an augmented mixed view through eyepieces of the viewing system. The mixed view may include graphical representations that are acquired or determined with information separate from the viewing system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation application of U.S. Serial No. 16/861,334, filedApr. 29, 2020, which includes subject matter related to U.S. Pat. App.Nos. 16/861,328 filed Apr. 29, 2020 and 18/168,296 filed Feb. 13, 2023.The entire disclosures of each of the above applications areincorporated herein by reference.

FIELD

The present disclosure relates to visualization, and particularly to amixed visualization during a procedure.

BACKGROUND

This section provides background information related to the presentdisclosure which is not necessarily prior art.

During a procedure, such as a procedure viewing the internal portions ofan object, various visualization and viewing systems may be used. Forexample, a microscope or visualization enhancement system may be used toview internal components of an object. The viewing system generallyallows for a direct and magnified viewing of the internal componentthrough various optical lenses during a procedure.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

A visualization system may include a microscope or other directvisualization system. The direct visualization system may include anoptical pathway that may include various optical lenses, such asmagnification, focusing, and the like. The lenses may be used totransmit light from an object to a viewing area that may include aneyepiece. A user of the microscope, which may define a focal plane, mayview the image through the eyepiece. The user may view or see the focalplane (also referred as a viewing plane) through the eyepiece.

In a viewing area, the user may view the object on which the procedureis performed. In the viewing area, a display may be provided thatdisplays the transmitted light from a work area for viewing by a user.Various augmentations may be provided in the viewing area and/or priorto the viewing area to augment the view of the object. The view providedto the user and/or view by the user may be a mixed view that includesboth a live view or image of the subject and other superimposed todisplayed features (e.g. graphical representation of hidden orobstructed objects).

In various embodiments, portions may be highlighted in the view,graphical representations of various items or portions and/or recalledand displayed visualizations may be overlaid on the current view of theobject. In various embodiments, for example, projections may be madeinto the optical path that illustrate or provide an augmented reality(AR), also referred to as a mixed view, that may be viewed by the user.Thus, the user may view both a real time display or view (e.g. liveview) of the object while also viewing an augmented or additionalportion (e.g. a graphical representation) displayed relative to the realtime image.

In various embodiments the object may be a living or non-living object,and may include a patient.

Further areas of applicability will become apparent from the descriptionprovided herein. The description and specific examples in this summaryare intended for purposes of illustration only and are not intended tolimit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only ofselected embodiments and not all possible implementations, and are notintended to limit the scope of the present disclosure.

FIG. 1 is an environmental view of a microscope system, according tovarious embodiments;

FIG. 2 is a detailed view of a microscope system and selected users;

FIG. 3 is a live view and selected information provided relative to thelive view with the microscope system;

FIG. 4A is a live view through a microscope system;

FIG. 4B is a superimposed view through a microscope;

FIG. 4C is a mixed live view and superimposed view through a microscopesystem;

FIG. 5A is a live view and mixed superimposed outline through themicroscope system;

FIG. 5B is a live view of a fluorescent through a microscope system;

FIG. 5C is a mixed view through a microscope system;

FIGS. 6A and 6B is a mixed live view and changing superimposed graphic;

FIG. 7 is a schematic illustration of a target object;

FIG. 8A is a first view at an initial time of the microscope system;

FIG. 8B is a live view and superimposed prior acquired image through amicroscope system;

FIG. 9A is a live view and mixed superimposed image through themicroscope system;

FIG. 9B is a mixed live view and superimposed graphical representationview of the microscope system;

FIG. 10 is a microscope system with a range finding portion; and

FIG. 11 is a process for displaying prior acquired information.

Corresponding reference numerals indicate corresponding parts throughoutthe several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference tothe accompanying drawings.

A procedure may be performed in a selected location, such as in asurgical suite 20. It is understood, however, that a surgical suite maybe any appropriate selected procedure room and a surgical suite ismerely exemplary. Further, it is understood that any appropriateprocedure may be performed relative to a living or non-living subject.For example, a procedure may occur on a complex machinery system wheredisassembling the entire system is undesired. Further, the procedure mayoccur on a living subject, such as a human subject, for selectedpurposes.

With reference to FIGS. 1 and 2 , in various embodiments, a surgicalsuite may include a viewing or optical system, such as a surgicalmicroscope system 30. The viewing system 30, which may also be referredto as a microscope herein, may include various components for performinga procedure on a subject 34. In various embodiments, the subject 34 maybe held or positioned with a support, such as a patient support oroperating table 38. The surgical microscope system 30 may include amicroscope component also referred to as a head assembly 32 that mayinclude one or more viewing ports or portals, also referred to aseyepieces or ocular lenses or units 44. The microscope 30 may be abinocular microscope such that a plurality, such as two ocular units, 44a, 44 b, may be provided. Additionally more than one ocular units, suchas a second or auxiliary ocular unit 48, which may also be stereoscopicor binocular, may be provided. Thus, more than one individual may viewthe same portion of the subject 34 simultaneously and view the subjectin three-dimensions due to the binocular system 44, 46.

During a selected procedure, the subject 34 may be viewed through theeyepieces 44. Positioned near or moved near the subject 34, and alsoformed or connected to the head portion 32, may be one or more objectivelenses 52. Therefore, the objective lens 52 may be positioned near thesubject 34. The object lens 52 gathers the light from the subject 34(e.g. reflected light) and allows transmission through the head portion32. Various lens systems in the microscope 30 may, therefore, allow foran enhanced or enlarged view of the subject 34 between the objectivelens 52 and the eyepieces 44.

In various embodiments, the surgical microscope 30 may include themicroscope head component 32, which may include optical components suchas eyepieces 44 and lens 52, and one or more arms or support members 60.The support members may include a first support arm 62 and a secondsupport arm 64. The second support arm 64 may be mounted to a floor ormain support 70 which may be fixed in a selected location and/or bemobile, via one or more wheels 72. It is further understood that thewheels 72 may be locked in a selected pose to fix the microscope 30 in aselected pose relative to the subject 34.

In various embodiments, one or more motors may be provided in and/or tomove the one or more arms 60. For example, a first motor 74 may beprovided to move the first arm 62 relative to the second arm 64 and asecond motor 76 may be provided to move the head 32 relative to thefirst arm 62. Thus, the head 32, including the lens 52, may be moved ina selected manner relative to the support member 70 and/or the subject34. The motors 74,76 may be appropriate motors, such as electric motorsthat may include encoders and/or operate with encoders to determineprecise movement (e.g. with a selected amount of movement or positionaldetermination error that may be within a selected tolerance, such asless than about 5 millimeters (mm), less than about 1 mm, less thanabout 0.1 mm, etc.). Motors 74, 786 may include DC stepper motors orother appropriate motors.

In various embodiments the motors 74, 76 may be controlled by a user 80.In addition, one or more control or processor modules or system 84 maybe provided to control the selected motors 74, 76 and/or othercomponents of the surgical microscope 30. For example, the processorsystem 84 may include one or more inputs 86 to control the microscopeportion 32 for selected movements and/or storing selected poses of themicroscope 32. Accordingly, after storage of selected poses of themicroscope 32, the microscope 32 may be moved back or returned to theselected poses, such as relative to the support member 70, for viewingafter a selected procedure of after a selected passage of time.

The processor system 84, therefore, may include a processor module 88and one or more memory modules 90. The processor 88 and the memory 90may be any appropriate processor or memory, such as those discussedfurther herein. Further, one or more display devices, such as an LCD orLED display device 94 may be provided for viewing various inputs,statuses, and/or images viewed with the microscope 32.

The user 80 may use or operate the microscope system 32 to perform aprocedure on the subject 34 with one or more instruments 110. Theinstruments 110 may include one or more tracking devices 112 that may betracked with one or more localization systems 114. The localizationsystems 114 may include one or more of an optical, electromagnetic,acoustic, or other localization systems that may be used to tracktracking devices and the portions to which the tracking devices areamounted. Generally, the localization system 114 may be used to track alocation of the tracking device 112 that may be associated with theinstrument 110 relative to the subject 34. Associated with the surgicalmicroscope system 30, including and or directly to the head component32, may also be a microscope or viewer tracking device 118. The subject34 may also have a tracking device 120 attached thereto. Therefore, theinstrument 110, the subject 34, and the microscope 32 may all be trackedrelative to one another, including simultaneously, with the respectivetracking devices and the tracking localizer and system 114. Otherappropriate pose systems may also be used to assist and/or performtracking such as tracking devices associated with the motors for movingthe microscope 32 and fixation of the subject 34 at a known pose. Asdiscussed above, encoders may be included with the motors 74, 76 and/orthat the joints of the members 62, 64 for determinate movement and/orpose of the microscope 32 such as in a robotic system or microscope. Invarious embodiments, the pose of the lens 52 relative to the subject 34may be most relevant for viewing various features relative to thesubject 34. As discussed herein, a pose of a portion (e.g. themicroscope 32, subject 34, and/or instrument 110) may include allposition and orientation information, e.g. six degree of freedom, suchas translational (x,y,z) coordinates and orientation (yaw, pitch, roll)coordinates.

In various embodiments, therefore, the pose of the instrument 110 may beillustrated as a graphical representation superimposed on a view throughthe eyepieces 44 of the microscope 32. Various tracking systems that areable to register prior acquired image or other data to a subject includethe StealthStation® S8 surgical navigation system sold by MedtronicNavigation having a place of business in Colorado and/or systemsdisclosed in U.S. Pat. App. Publication No. 2019/0328461, published Oct.31, 2019, incorporated herein by reference.

Registration may include registering prior acquired information (such asimage data) to the subject 34 for determining a pose relationship of theprior acquired image data to the subject 34 in real time. Similar, thetracking systems may be used to illustrate the pose of tracked systemsto the prior acquired information.

In various embodiments, the system including the localizer 114 mayincorporate various portions or systems, such as those disclosed in U.S.Pat. Nos. RE44,305; 7,697,972; 8,644,907; and 8,842,893; U.S. Pat. App.Pub. No. 2004/0199072, and U.S. Pat. App. Pub. No. 2019/0328460 allincorporated herein by reference. The localizer 114 of a navigationsystem may be used to track a pose of an object, as discussed herein.The pose may then be displayed for viewing by the user 80, as alsodiscussed herein. Also, tracking information, including informationregarding the magnetic fields sensed with the tracking devices and/oroptical signals may be delivered via a communication system to selectedportions of the microscope system 20, such as the processor 88. Thecommunication may be wired or wireless using appropriate systems.

Image data, in addition to the live view, as discussed herein, may beacquired such as with one or more of the imaging systems prior to and/orduring a surgical procedure for displaying an image or portion thereoffor viewing by the user 80 through the eyepieces 44. The additionalinformation or data may be viewed by the user 80 with the live view, asdiscussed herein.

A navigation or tracking domain or volume generally defines a navigationspace or patient space in which objects may be moved that are tracked,such as with the localizer 114. Due to the tracking devices 112, 118,120 the various portions may be tracked relative to one another. Thenavigation volume or patient space may be registered to an image spacedefined by the additional information (e.g. prior acquired image orselected image) of the subject 34. The registration allows forillustrating and displaying determined positions of various objectsrelative to one another and/or the image space. In various embodiments,registration may occur by determining identical points or portions (e.g.registration points or landmarks) in the image space and the navigationspace. The registration points may be natural (e.g. anatomical) orartificial (e.g. fixed to the subject). Once determined in both spaces,however, a registration may be made by determining a transformationbetween the two spaces using the registration points. The transformationand registration may be carried out by executing instructions with anappropriate processor, such as the processors 88. Registration of thepatient space to the image space and determining a position of atracking device, such as with the tracking device 118, relative to aDRF, such as the DRF 120, may be performed as generally known in theart, including as disclosed in U.S. Pat. Nos. RE44,305; 7,697,972;8,644,907; and 8,842,893; and U.S. Pat. App. Pub. No. 2004/0199072, allincorporated herein by reference.

With continuing reference to FIGS. 1 and 2 and additional reference toFIG. 3 a view through the eyepieces 44 may include a display or image130. The image 130 may include or display various information such as adirect or live view, also referred to as a real time view, 132 of aselected portion of the subject, such as a brain 134. The live view 132may be viewed directly through the microscope 32 such as a generaldirect light transmission through the eyepieces 44 for viewing by theuser 80. The live view may be of a viewing area or procedure area at aviewing plane. The viewing plane is generally understood to be the areaor plane at which the microscope is focused and the user 80 may viewthrough the eyepiece 44. The live view may be a current or real timeview.

In addition to the live view 132 in the view 130, additional informationmay be superimposed onto the live view 132 and/or displayed near oradjacent the live view due to, for example, tracking of the subject 34with the subject tracking device 120 and/or the microscope 32 with themicroscope tracking device 118. In various embodiments, one or moretargets may be determined. The one or more targets may be referred toherein as one or a target, but it is understood that more than onetarget may be determined. Also, the target may be any appropriateportion, such as an anatomical feature, an anomaly, etc. and/or aselected location (e.g. a volume include a selected specific feature,such as a tumor). For example, acquired image data, such as MRI imagedata, may be used to identify a tumor as the target and an outline of atumor that may be displayed as a target graphical representation 140. Inaddition, the instrument 110 may be illustrated as an instrumentillustration or graphical representation (also referred to as an icon)144 due to the tracking of the subject 34 and the instrument 110. Thus,a pose of the instrument 110 may be viewed directly with the microscope32 through the eyepieces 44 and/or a representation of the instrument110 may be illustrated as the graphical representation 144.

As illustrated in FIG. 3 , through the eyepiece 44, in addition to thelive image 130, an additional overlay or graphical view 150 may bedisplayed. The overlay 150, or any appropriate graphical representation,may be opaque, transparent, or at least partially transparent. In apartially transparent overlay, the live image may be viewable throughthe overlay 150. The mixed or augmented view 150 may include additionalor selected information such as the graphical representation of theinstrument 144 and/or the representation of the tumor or selected target140. The augmented view 150 may additionally include information, suchas a depth line or indication 154 from a surface 158. The plane orsurface representation 158 may be a surface of the brain 132, or otherselected portion of the subject 34 in the live view 130. The surface mayalso be a view plane through the eyepiece 44

As discussed above, the pose of the subject 34 may be determined withthe subject tracking device 120 which may include a pose of selectedportions of the subject, such as a surface of the brain 132. Also, thesubject 34 may be fixed in a selected position with a holding orfixation device, such as a head clamp 39. Therefore, the outer orsurface line 158 may be determined and illustrate the augmented view150. The instrument 110 may be tracked with the tracking device 112 andillustrated by the graphical representation 144 in the augmented view150.

Additionally, information regarding the subject 34, such as fibertractography or tractography representations 162 may also be displayedin the augmented view 150 relative to the live view 130 of the subject34. The fiber tracts may be determined with additional, including,previously acquired information. The prior or additionally acquiredinformation may be registered to the subject due to the tracking of thesubject 34, as discussed above. Further, the pose of the microscope 32may be known relative to the subject 34. Thus, the additional registeredinformation (e.g. tractography) may be displayed in the mixed view.Thus, the user 80 may view the subject 34 through the microscope 32 andview additional information, in addition to the live view 130 of thesubject 34, including the augmented view 150. The outline of the target140, a graphic representation of the instrument 144, and selected depthor target information may also be displayed. This allows the user 80 toview a depth or three-dimensional representation of the selected targetat its location within the subject through the microscope view ports 44simultaneously with a live view. The augmented image 150 may allow forthe user 80 to better understand a representation of the view or imageof the subject 34 rather than viewing simply or only the live viewthrough the microscope 32.

As discussed above the live view 130 as viewed through the eyepiece 44allows the user 80 to directly view a live view or a current view of oneor more portions of the subject 34. The additional data, such as thegraphical representations of the tumor or target 154, graphicalrepresentation of the instrument 144, representation of a surface orouter extent 158, and/or tractography information 162 may be based uponvarious prior acquired image data and/or analysis of the subject 34. Thepositioning of the graphical representations in the live view and/or onthe additional or augmented view 150 may be based upon tracking thesubject 34 with the subject tracking device 120 and the microscope 32with the microscope tracking device 118.

The additional information may be registered to the subject 34 due to aselected registration, such as image registration, to the subject 34 asis generally understood by one skilled in the art. For example, thetracking localizer 114 may be used to track a plurality of portions,such as the subject 34 with the subject tracking device 120 and/or themicroscope 32 with the microscope tracking device 118. One or moreselected processors, such as the processor 88, may be used to perform aregistration of the subject space defined by the subject 34 relative tothe microscope 32 and/or additional image data space. The trackingsystem may be one similar to the Stealth Station® Navigation System,sold by Medtronic Navigation, Inc. having a place of business inColorado. The various navigation systems or tracking systems may be usedto determine the pose of the subject 34 relative to the microscope 32with the respective tracking devices and, therefore, determine the poseto place the graphical representations, such as the graphicalrepresentation of the instrument 144, in the augmented view 150. Theaugmented view may be displayed with a semi-transparent screen or plate,such as those included in the Kinevo 900 microscope, sold by Carl ZeissMeditec AG having a place of business in Germany and/or a microscopehaving a screen of a selected resolution, that may be used to displaythe augmented view 150. Further, the user 80 may select to determine oridentify graphical representations to display in the augmented view 150such as with the inputs 86 with the microscope system. Accordingly, theuser 80 may select to display or not display various items, such as thesurface line 158, the selected outline 140, or other selected graphicalrepresentations.

Additionally the various graphical representations may be displayed inappropriate manners such as in two-dimensional representations,three-dimensional representations, or in changing representations suchas a time varying may be based upon various information, such as gatingrelative to a heartbeat of the subject 34. The three-dimensionalrendering may be displayed to the user 80 as a three-dimensional imagedue at least in part to the binocular eyepieces 44. Thus, the user 80may view the image of the subject 34 augmented with selected graphicalrepresentations, as discussed above. Particularly the augmented view 150may display or illustrate a depth from a selected surface, such as atthe surface line 15. The augmented display may also illustrate aposition or depth of the instrument 110, such as with the graphicalrepresentation of 144, including viewing a distal tip 144 d of theinstrument 110.

With continuing reference to FIG. 1 and FIG. 2 , and additionalreference to FIGS. 4A, 4B, and 4C, a live view 170 through a microscope32 is illustrated. The live view 170 may allow for viewing selectedportions of the subject 34, such as a surface of a brain 174. Inaddition to the surface of the brain 174, various other items may alsobe viewed, such as cranial bone 176 and the instrument 34. The eyepieces44 allow for direct visualization of portions of the subject 34, such asthe instrument 110 and/or portions of the subject 34, such as the brain174 in the live or real time view 170. The direct visualization by theuser 80 may be of any items within the visual range of the microscope 32relative to the subject 34.

As discussed above, the subject 34 may be tracked with the subjecttracking device 120 and the microscope 32 may be tracked with themicroscope tracking device 118. The instrument 110 may be tracked withthe instrument tracking device 112. According to various embodiments, adisplay, such as a mixed or superimposed view through the eyepieces 44,includes a graphical representation or a graphical overlay in athree-dimensional manner, as illustrated in FIG. 4B. The graphicalrepresentation may include a three-dimensional graphical representation180 of the instrument 110 relative to portions of the subject, such asof a graphical representation of a brain surface 184 and arepresentation of selected portions or targets 188, such as anidentified tumor. Additionally, various anatomical features may also beillustrated in the graphical representation such as vessel structures192 and/or other anatomical structure of features such as ventricles,fiber tracts, optical nerves, eye balls.

Thus, the display through the eyepieces 44 may be of a three-dimensionalrepresentation based upon various information, such as prior acquiredimage data of the subject 34 and/or graphical representations of theinstrument 110, such as the graphical representation 180. Thepositioning of the various representations, including of the selectedtarget or tumor 188 and/or the instrument representation 180 may bebased upon tracking the subject 34 with the subject tracking device 120and the instrument 110 with the tracking device 112. Accordingly, athree-dimensional representation may be displayed for viewing by theuser 80 with the microscope 32 including prior acquired information orinformation not viewable directly with the microscope 32. While themicroscope 32 may provide or allow a magnified view that is a live viewof the subject 34, as illustrated in FIG. 4A, the representation ofadditional information for display relative to the subject 34, such assuperimposed and/or displayed with the live image 170, may allow foradditional conceptualization or understanding of the subject 34 and/orportions relative thereto by the user 80.

During the surgical procedure, or any appropriate selected procedure,the user 80 may select to display or not display various information,including the three-dimensional representation as illustrated in FIG.4B. The user may use the input 86 or instruct the inputs 86 (e.g.instructing an assistant, voice controls, or the like) to display or notdisplay various three-dimensional features.

Additionally, and/or alternatively, two-dimensional representations maybe displayed superimposed and/or overlaid on the live image 170, asillustrated in FIG. 4C. The live image 170 may display various portionsof the subject 34 as illustrated directly or viewed directly through themicroscope 32 by the user 80. Thus, the live view may display variousportions, such as the bone structure 176 and/or other features. However,displayed or superimposed on the live image may be representations ofvarious structures and/or features, such as a prior identified tumor ortarget 200 that may be displayed, such as by an outline or indication.Further, various anatomical structures, such as vasculature or skeletalmay also be displayed by overlays, such as anatomical overlays 204.Again, the overlays may be displayed relative to the live image 170 dueto a registration and/or tracking of the subject 34 and the microscope32 and registration to the prior acquired image. In various embodiments,the user, or other appropriate user, may identify a selected featuresuch as anatomical features and/or targets (e.g. a tumor) in prioracquired image data (e.g. magnetic residence imaging (MRI)). The prioracquired and/or determined features may then be registered to thesubject 34 during a selected portion of a procedure and superimposed onthe live image 170, as illustrated in FIG. 4C. This allows the user 80to understand or recognize poses of identified features and alternativeor additional image (e.g. MRI) for display on the live image 170 of thesubject 34. Thus, the user 80 may view both a live image and/or theaugmented image to understand the additional features identified inalternative image data.

Again, the user may determine to display or not display various featureswith the inputs 86 of the microscope system 30. The user may use themicroscope 32 to directly view the subject 34 and/or augment the view ofthe subject 34 during a selected procedure. The additional informationmay be used by the user 80 to identify and/or understand representationsor poses of previously identified or determined features, such as tumorsor selected anatomical targets or features.

With continuing reference to FIG. 1 and FIG. 2 , and additionalreference to FIGS. 5A, 5B and 5C, the display or image view by the user80 may include a substantially live view 220 of the subject 34 includinga surface of a selected portion of the subject 224 such as a brainsurface. As discussed above, various additional information may besuperimposed and/or displayed on the live image 220, such as an outline228 of a selected target or portion (e.g. a tumor). The target imagedisplay 228 may be based on prior acquired or reviewed information, suchas MRI data.

During a selected procedure, additional procedures or information may beacquired or utilized. Various digital techniques and/or operativetechniques may include fluorescence of selected anatomical features orportions of the subject 34. For example, as illustrated in FIG. 5B, afluorescent material and/or materials may be provided to the subject 34.Thereafter, the materials may fluoresce and/or may be induced tofluoresce in a selected manner. As illustrated in FIG. 5B, for example,a selected area may have a first fluorescent 230 and a second area mayhave a second fluorescent 234. The image or view may be the live view220 of the subject 34 through the microscope and the user 80 may viewthe selected fluoresce directly. Additionally, the image may be viewedor analyzed by the microscope system 30. The system may identify ordistinguish between the high or bright fluorescence 230 and the lowfluorescence 234. Based upon differentiation of the fluorescence, theprocessor system 88 may identify or segment the bright area 230 from thedarker area 234. In various embodiments, a high or bright fluorescence230 region may be referred to as a bright region and may have abrightness or luminescence that is at least about 10% brighter whencompared to the low fluorescence 234 region. In various embodiments, thebright region may have a brightness or luminescence at least about 50%brighter, at least about 200% brighter, or about 400% brighter. Thebright region may be a two-dimensional area and/or a three-dimensionalvolume.

The microscope system 30, including the processor, may then segment theview or image of the subject 34 based on the fluorescence. The area orline between the bright fluorescence 230 and the low fluorescence 234may be determined in any appropriate manner, such as by segmentation.The segmented region, therefore, may be used to determine a current orupdated region of volume of the target. The updated region may then bedisplayed, as discussed herein.

As illustrated in FIG. 5C, therefore, the live view 220 may haveoverlayed thereon a mixed or augmented outline graphical representation240 that may be based upon the segmentation of the bright fluorescencearea 230 from the darker fluorescence area 234. In various embodiments,for example, the second or augmented outline 240 may be displayed withthe fluorescence and/or after the fluorescence has ceased. Thus, it isunderstood, that the secondary or augmented for representation 240 mayalso be displayed without the fluorescence. For example, during aselected procedure, fluorescence may occur during illumination of thesubject 34 with a selected wavelength of light or energy. At that time,the fluorescence may occur within the subject 34 and the image may beanalyzed by the microscope system 30. After analysis of the image, theidentification of the augmented outline 240 may be made. The outline maythen be displayed for view by the user 80, such as an overlay orsuperimposing of the outline 240 on the live view 220.

The augmented or updated outline may be used by the user 80 to determinea progress of a selected procedure. For example, during a selectedprocedure an ablation or removal of a selected portion of the subjectmay occur, such as removal of a tumor or removing or ending an aneurism.Thus, the user 80 may view the updated outline 240 to assist indetermining progression of the procedure. The outline, or anyappropriate representation, may therefore allow for a geometry (e.g.shape and size) to be illustrated and changes relative to the initial orfirst outline or representation 228. This may be in addition to a pose(i.e. x,y,z location and orientation) thereof.

It is understood that the luminescence of the subject 34 may occur morethan once and may occur at various times during a procedure. Thefluorescence may be used to identify or determine an update or change inthe subject anatomy or selected portion thereof. The user, therefore,may understand or better visualize a change in the subject or portion ofthe subject 34.

In various embodiments, the wavelength of illumination of fluorescentsof the selected portion of the anatomy may be in a non-visiblewavelength. Accordingly, the fluorescence that may be a brightfluorescence 230 relative to a darker fluorescence 234 may not bevisually identifiable by user 80. The microscope system, or appropriatesystem, however, may analyze the image of fluorescence in thenon-visible wavelength to identify the differentiation between portionsof the anatomy or subject. Thus, the outline 240 that may besuperimposed on the live image 220 may be the only visual representationto the user 80 of a differentiation of fluorescence. Accordingly, themicroscope system 30 may be used to display a difference between aninitial outline 228 and an updated or second or later outline 240. Asdiscussed above, the initial outline 228 may be based upon prioracquired image or information of the subject 34 such as with image dataincluding MRI image data. A change in the subject or portion of thesubject during a procedure may be identified and determined anddisplayed (e.g. with an augmented outline) for use by the user 80.

With continuing reference to FIGS. 1 and 2 and additional reference toFIGS. 6A and 6B, a live view image 260 of the subject 34 may includeviewing a portion of the subject, such as a brain surface 264 directly.Further, an outline or graphical representation 268 of a selectedportion or target, such as a tumor, may also be displayed. The targetmay be determined or selected in an image or live view in variousmanners such as by manual determination (e.g. a user outline thetarget), image segmentation (e.g. automatic segmentation based onselected algorithms such as color or edge detection), and/or recalling aprior determined or delineated target (e.g. accessing and recalling froma memory). As discussed above, the identification of the tumor as thatis illustrated as the graphical representation 268 may be based on prioracquired information, such as prior acquired MRI data of the subject 34.The graphical representation 268 may be based upon a registration of theprior acquired image and/or determination of a tumor or target to thesubject 34 in part due to tracking the subject 34 with the subjecttracking device 120. As also discussed above, additional graphicalrepresentations may be displayed relative to the live image 260, but arenot displayed here for clarity of the current discussion.

The graphical representation 268 may illustrate a selected or determinedboundary of the tumor relative to the live image 260. The live image 260may be viewed by the user 80 through the eyepieces 44. Accordingly, evena binocular three-dimensional view of the brain surface 264 may beaugmented with the graphical representation 268 to illustrate a depth,shape, and the like of the selected target. In addition to a staticboundary, however, a pulsing or moving boundary representation 272 maybe illustrated. With continuing reference to FIG. 6A and additionalreference to FIG. 6B, the live view 260 may illustrate a substantiallyunchanging or unmoving portion of the subject, such as viewing the brainsurface 264 in a fixed location. Also, the static boundary 268 of theselected target or tumor may also be displayed relative to the surface264 of the brain. The pulsing or changing representation 272 may,however, move or appear to move such as changing a highlighted portionor color of the boundary of the tumor over a period of time. Forexample, the pulsing shell or grid 272 in FIG. 6A, may illustrate anupper most or closest to the surface boundary of the tumor. At a secondtime, such as over a period of time of a fraction of a second ornumerous seconds (e.g. about 0.1 to about 0.5 seconds) the pulsing grid272 may change shape and position to a second grid position 274, asillustrated in FIG. 6B.

At a selected time, the user 80 may select to have a geometry of thetumor or target of the subject 34 illustrated in a changing or pulsingmanner. Accordingly, a first portion of the pulsing grid 272 may behighlighted or illustrated, in FIG. 6A. In the same view or plane, achange in the highlighted portion of the grid or outline of the tumor,as illustrated by the different or second portion 274 in FIG. 6B. Thisallows the user 80 to view a dynamic change in a selected portion of theview without moving the microscope 32.

It is understood that any appropriate number of highlighted portions maybe illustrated over a selected period of time, such as in sequence. Forexample, the tumor or target, having the static graphic representation268, may have a selected depth or exterior geometry, as exemplaryillustrated in FIG. 7 . During a time zero or at a time zero (T0), thegrid shell 272 may illustrate the boundary at a surface or zeromillimeter depth into the tumor or target 268 i that is graphicallyrepresented by the outline 268 in FIGS. 6A and 6B. At selected timeperiods, such as about 0.1 to about 0.5 seconds apart, an outline atselected depths, such as 1 millimeter depths, may be illustrated byhighlighting or changing a color of the grid, for example as illustratedby the different grid portion 274 and illustrated in FIG. 6B. It isunderstood that any appropriate number of dimensions or times may beused and 1 millimeter dimensions at selected times T0 through T5, asillustrated in FIG. 7 , is merely exemplary. Nevertheless, at theselected times, for example, the different portions at T0 through T5 maybe sequentially highlighted and illustrated in the display, such assuperimposing on the live view 260 for viewing by the user 80.

As the selected portion or different portions of the target, such as thetumor 268 i are highlighted, the user 80 may understand the outline ofthe tumor 268 i at a different depth relative to the surface or zeromillimeters. Thus, the user 80 may understand an outline or shape of thetumor 268 i when viewing the live image 260 with the microscope 32. Thepulse geometry outline 272, 274 may be initiated or stopped by the user80 at selected times to understand the predetermined or identified shapeof the tumor 268 i. In other words, the user 80 may update the portionor location of the pulsing (e.g. a pulsing plane) to be highlights ordesignated. Thus, rather than an automatic pulsing, the user 80 mayselect the portion or plane to be highlighted. The outline or pulsingshape may be displayed for understanding of the target by the user 80within the subject 34.

With continuing reference to FIG. 1 and FIG. 2 and additional referenceto FIGS. 8A and 8B, the microscope 32 may be used to view a live image300 of the subject 24, as discussed above. The live image 300 mayinclude various portions, such as a surface 304 of a brain of thesubject. In addition to a surface of an organ, such as the brain 304,other features may also be viewed or displayed by viewing through theeyepieces 44 of the microscope 32. For example, a bone structure 308 maybe viewed and one or more other soft structures or soft tissues, such asa vasculature including one or more arteries 312. Although thediscussion herein, for example with reference to FIGS. 8A and 8B, mayrefer to the artery 312, it is understood that various features may bedisplayed and analyzed in addition to the artery 312 as discussedfurther herein.

As discussed above, the microscope 32 may be fixed relative to thesubject 34 and/or tracked relative to the subject 34. In variousembodiments, for example, the base stand 70 may be fixed relative to thesubject 34 and the subject 34 fixed relative to the microscope 32, suchas the head clamp or holding frame 39. Accordingly, the microscope 32may move relative to the subject 34, such as for obtaining access toportions of the subject 34, viewing more than one area of the subject34, or other appropriate reasons. As the microscope 32 moves, theeyepiece 44 also moves. Generally, the view of the user 80 is with theeyepiece 44 and movement of the eyepiece 44 may be determined or mostrelevant relative to movement of the microscope 32. Thus, movement ofthe microscope 32 may be understood to refer or relate to movement ofthe eyepiece 44. Regardless, the microscope 32, including the microscopesystem 30, may be used to obtain images of the subject 34 at varioustimes, such as at a first time and a second time.

During the surgical procedure, the microscope may be moved from a firstor original pose at a first time, to a second pose at a second time andthen back to the original or first pose at a third or selected time. Thevarious poses of the microscope 32 may be determined due to tracking themicroscope 32 with the selected tracking device, including a microscopetracking device 118 and the patient tracking device 120. Further, asdiscussed above, the selected arms or mechanical robotic portions of themicroscope system 30, including the arms 62, 64 may be moved withsubstantial precision due to the selected motors 74, 76. Regardless, themicroscope 32 may be moved from a first pose or an original or forpurposes of viewing the subject 32 after a selected period of time.

Accordingly, for example, as illustrated in FIG. 8A, a first view at thefirst pose of the microscope 32 of the subject 34 may be directly viewedthrough the microscope 32. At that time, an image or snapshot may beacquired of the subject 34 through the microscope 32. In variousembodiments, a camera or image gathering system may be associated withthe microscope 32. The image may be stored in a selected memory system,such as the memory 90, for recall at a later time. The image may includevarious portions, such as the surface 304 of the brain, the bone 308,and the artery 312. Snapshots at selected periods of time, including afirst time T1 and a second time T2 may be displayed relative to the rowview or live view 300.

With reference to FIG. 8B, therefore, after a selected period of time,such as at a time T3, the microscope may be returned or moved to thepose at time T1. The pose at the T1 time may be determined or saved dueto the tracking of the microscope 32. In various embodiments, therobotic arms (including the motors 74, 76) may be used to automaticallyor upon an input command to move the microscope to a prior pose (i.e. apose at T1 at which an image was acquired for a snapshot). Accordingly,a live view 320 may be of the subject 34, including a brain surface 304a. Overlaid on the live view 320, including the brain surface 304 a maybe an image acquired at time T1, as illustrated in FIG. 8A. Due tovarious analysis of the image and/or identification by the user 80,various vasculature or portions may be identified in the image overlaidon the live view 304 a. For example, as illustrated in FIG. 8B, theoriginal pose of the vasculature 312 may be displayed relative to a newor changed pose 324. Further, the microscope system 30 may includeindications or arrows 328 to illustrate determined orientation or changein pose of the subject 34, such as the brain surface 304 a at thecurrent time relative to the original time for which the overlaid viewis made. The arrows 328 may include an original at the original or T1pose of the selected structure (e.g. vasculature 312) and a head or tipat the current or live view pose of the selected structure (e.g.vasculature 324).

Any appropriate number of prior time snapshots may be displayed oracquired. For example, a plurality of snapshots 340 may be generated atselected times, such as time T1, time T2, and time T3. Each of thesnapshots 340 may be displayed overlaid on the live view of the currentsurface of the brain 304 a for viewing by the user 80. Accordingly, theuser 80 may select one or more of the snapshots 340 for displaying toaugment the current live view of the surface of the brain 304 a.

The prior snapshots may be overlaid on the current live view due to thepose of the microscope 32 back at the pose at a prior time, such as thetime T1. Accordingly, the change or possible change of selected portionsof the anatomy, such as a vasculature, may be viewed and displayedrelative to an initial view or a first view acquired during a movementof the microscope 32 at a prior time or first time in a procedure.Regardless, the user 80 may view the live view with an overlay orsuperposition of a prior acquired snapshot for viewing the results ofthe change in time and/or a portion of a procedure relative to thesubject 34.

With continuing reference to FIG. 1 and FIG. 2 , and additionalreference to FIGS. 9A and 9B, the user may view the subject 34 with themicroscope 32 such as to view a live or real time view 420 of thesubject 34. As discussed above, the microscope 32 may be moved by theuser 80 in an appropriate manner, such as from a first pose to a secondpose. Further, the microscope 32 may be tracked with the tracking systemincluding the localizer 114 microscope tracking device 118 relative tothe subject 34 with the subject tracking device 120 or due to variousmovement determining portions such as those included with the motors 74,76 and/or included robotic features. In various embodiments, therefore,the user 80 may determine or track a pose or know a pose of themicroscope 32 relative to the subject 34. In addition, the user 80 mayreturn the microscope 32 to a prior pose by tracking or saving a pose ofthe microscope 32 relative to the subject 34. Therefore, the microscope32 may be used to view the subject 34 at a first time and a second timeand make comparisons between the two times, for example as discussedabove. In addition thereto, however, the user 80 may also determine orknow the pose of the microscope for viewing the subject 32 and analyzingor evaluating poses of selected portions of the subject 32 at differentperiods of time or over a period of time.

With reference to FIG. 9A, for example, the user 80 may identify variouscheckpoints or landmarks in the subject 34. For example, with referenceto FIG. 9A, the user 80 may identify one or more physical or reallandmarks in a live view or view 420 through the microscope. A physicallandmark may include a marking (e.g. optical marking), such as an X orink marking including a first physical landmark 424, a second physicallandmark 428, a third physical landmark 432, and a fourth physicallandmark 436. The physical landmarks 424-434 may be identified by theuser 80 such as identifying a physical landmark in the subject, orplacing a physical landmark on the subject, such as the ink markings.The landmarks may be identified by the user 80, such as identifyingportions that are illustrated or viewable in an image.

The microscope system 30 may identify the physical landmarks such asthrough an analysis of the image viewed with the microscope 32. Ananalysis may include a machine learning algorithm to identify selectedfeatures of the physical markings, or other appropriate identificationor determination measures. For example, as illustrated in FIG. 9A, theintersection of the two ink marks (i.e. “X”) may be used as anindication of a particular finite or fixed checkpoint 424 i-434 i. Thecheckpoints may be identified in or to the microscope system 30, such asthrough an automatic identification, manual identification (e.g.identification by the user 80 to the microscope system 30 with variousinputs, including the input 86), or other appropriate determinations.

Further, as discussed above, the microscope system 30 may acquire ordetermine an image of the subject 34 at a first time. After a selectedperiod of time, such as after one minute, two minutes, or after aselected time based upon a decision of the user 80, a second image ordetermination of the checkpoints 420-434 may be made. As discussed abovethe pose of the microscope 32 may be known or determined with thetracking system, including the microscope tracking device 118 relativeto the subject 120 and/or due to the robotic or motor mechanism 74, 76in the microscope system 30. Thus, the microscope 32 may be moved toview the subject 34 at substantially the same pose as during theacquisition of an initial determination of the checkpoints 424-434. Thisallows an analysis may be made between the initial and currentcheckpoints.

As illustrated in FIG. 9A, the current checkpoints may be viewed as thephysical markings and the prior checkpoints may be indicated asgraphical representations, such as by “CP” followed by a number.Further, various indications may be provided regarding the initialcheckpoint pose such as a color, indication, or a key, as illustrated inFIG. 9A. Again, the key may be an augmented view through the eyepieces44 relative to the image 420 displayed for view by the user 80.

Regardless, for example, CP1 may be illustrated in green and the key mayindicate that less than one millimeter distance change has been made ordetermined between a prior pose determination and a current posedetermination. Alternatively, or in addition thereto, the CP3 may beindicated as “off plan” or greater than a predetermined or set limit.The additional checkpoints CP3 and CP4 may also be illustrated inselected colors, such as yellow and red respectively, relating to thedistance of change between the initial illustration or determination andthe current pose. Thus, the microscope system 30 may be used toillustrate an amount of change in pose of the subject 34 from a firsttime to a current time. In various embodiments, this may be used todetermine if a re-registration is necessary and/or may be determinedwith a re-registration of re-alignment of the microscope.

In various embodiments, for example, the checkpoints may be placed onsubstantially rigid structures, such as a bone structure 450 of thesubject 34. Thus the rigid bone structure may be determined to havemoved between a first time and a current time and a re-registration orredetermination of a pose of the subject 34 may be indicated.

Further, various soft tissue or organ poses may also be determinedrelative to selected checkpoints. For example, with reference to FIG.9B, the checkpoints 424-434 may be illustrated relative to a soft tissue460 of the subject 34. Further, various soft tissue portions, such asfunctional indications may be illustrated including a first functionpose 464 and a second function pose 468. The function poses 464, 468 maybe illustrated or marked on the brain 460 with an appropriate markingsuch as a bio-compatible marker or physical portion. Additionally, dueto a determined or trackable pose of the microscope 32 relative to thesubject 34, the functional poses 464, 468 may be illustrated assuperimposed portions or features on the brain 460 for viewing by theuser 34 in the view 470. Thus, the live view 470 may have superimposedthereon indications of the functional poses 464, 468.

The functional poses 464, 468 may be determined due to probing of thebrain 460 by the user 80. Accordingly, the functional poses 464, 468 maynot include physical or identifiable landmarks on the brain 460, but maybe based upon functional indications determined by the user 80.Nevertheless, the microscope system 30 may view the functional poses464, 468 relative to the checkpoints 424, 434. The functional poses 464,468 may be displayed as graphical representations to the user 80,however.

A determination of movement or lack of movement may be used to determinewhether there has been brain shift of the brain 460 relative to thefixed or rigid portions, such as bone 450, of the subject 34. Forexample, the microscope system 30, in a manner similar to that discussedabove, may identify or determine the checkpoint poses 424-434. Thefunctional poses 464, 468 may then be determined relative to thecheckpoints 424-434 at a first time and a second time, or a time laterthan the first time. An indication of movement or no movement,therefore, may also then be made to the user 80 in the view.

For example, relative to the live view 470, an augmented portion,including a key, may be viewable by the user 80 through the eyepieces44. For example, the numbers 1 and 2 may be used to identify thedifferent functional poses 464, 468, respectively, and an indication ofwhether they are on or a distance relative to an initial pose may bemade. The key may indicate that functional point 1 is on the same orsimilar pose and functional point 2 is at a distance greater than twomillimeters relative to the initial pose. Thus, the user may identify orunderstand the current pose, including, if present, amount of change tothe current pose of the brain 460 relative to the checkpoint between twoperiods of time. Again, therefore, the user 80 may then determinewhether a procedure may continue or be augmented due to a possible shiftof the brain 460. Further, a determination of a pose of the functionalpoint may be reevaluated based upon indication of possible change.

With continuing reference to FIG. 1 and FIG. 2 and additional referenceto FIG. 10 , the microscope may include additional or alternativetracking or locating the systems in addition to the tracking device 118associated with the microscope 32, as illustrated in FIG. 2 , or otherportions. For example, a relative range finding system may include oneor more range finders such as a first range finding system or portion500 and a second range finding system or portion 510. Each of therespective range finding portions 500, 510 may project or include rangefinding beams or features 514, 520, respectively. The beams 514, 520 maybe signals (e.g. light or acoustic signals) that are sent and receivedfor range finding. The range finding system may be any appropriatesystems, such as determining a linear distance or determining athree-dimensional (3D) scan and related distances, and/or etc.

The range finding features 514, 520 may be emitted toward the subject 34that is supported on a selected support structure, such as the support38. The range finding features 500, 510 may be incorporated into themicroscope 32 and/or mounted to the microscope 32. Accordingly, therange finding portions 500, 510 may be removably mounted to themicroscope 32 for a selected procedure and/or portion of the procedure.

The range finding features 514, 520 that may be emitted and/or receivedby the range finding portions 500, 510 may be selected range findingfeatures such as optical range finding features. In various embodiments,for example, the range finding portions 500, 510 may form or beincorporated into a binocular or three-dimensional imaging range findingsystem that may triangulate a pose (including distance) relative to therespective range finding portions 500, 510 and the microscope 32 towhich they are fixed. In various embodiments, for example, the rangefinding portions 500, 510 may generate or develop an image of thesubject 34, such as a brain portion thereof, to determine a pose of themicroscope 32 relative to the subject 34 (and/or a portion thereof), ordetermine movement of the subject 34, such as a brain therein, relativeto the microscope 32 due to a known or fixed pose of the microscope 32relative to a portion of the subject 34 and/or the support 38.

The ranging or range finding portions 500, 510 may also be otherappropriate range finding features such as optical (visible andinfrared), acoustic, an ultrasound ranging system, radar ranging system,or other appropriate ranging system. Regardless, the ranging system maybe incorporated into the microscope 32 and/or affixed to the microscope32 for a selected procedure or a portion of a procedure. Thus, theranging system 500, 510 may be used to identify pose of the microscope32 relative to the subject 34 and/or pose of portions of the subject 34relative to other portions (e.g. brain shift).

In various embodiments, the ranging system portions 500, 510 may emitthe respective beams 514, 520. The beams 514, 520 may reflect off of thesubject 34 or portions thereof. The reflected portions may be receivedby the ranging system portions 500, 510 or other appropriate portions.The emission and receiving may be used to determined pose of the subject34 relative to the microscope 32 including the ranging system portions500, 510. As noted above, the pose of the instrument 110 may also bedetermined relative to the microscope 32 and the subject 43.

Further, the ranging system portions 500, 510 may include variousfeatures such as voice activation or controls. For example, the user 80may provide an audible input or command to range or determine an amountof brain shift, pose of the microscope or pose of a view relative to themicroscope relative to the subject 34, or the like. Thus, the user 80may interact and provide selected input to the range finding system 510,520.

Also, the microscope 32 may also view the instrument 110 that may bemoved relative to the subject 34. Thus, the ranging system 500, 510 maydetermine a pose of at least a portion of the instrument 110 relative tothe subject 34 and be able to view or display the information in theview through the view finders 44 of the microscope 32. Accordingly, therange finders 500, 510 may be used to range find relative to selectedportions of the subject 34 and/or the instrument 110 relative to themicroscope 32. The information may be displayed for viewing by the user80 in the view ports or eyeholes 44, such as for determination of a poseor movement of the subject 34 and/or pose or determination of a pose ofthe instrument 110.

Accordingly, as discussed above, the microscope system 30 may be used toprovide information to the user 80 to assist in a selected procedure.Pose of instruments, such as the instrument 110, and/or overlays may bedisplayed in the eyepieces 44 for viewing by the user 80. Thesuperimposed information (e.g. target location or dimensions, trackedinstruments) may be viewed by the user 80 while the user 80 views thesubject 34 through the eyepieces 44. Thus the user 80 may include a viewof the subject 34 simultaneously with additional information that may beacquired separately from the microscope 32 and/or prior to use of themicroscope 32. The microscope system 30 may allow for the user 80 toprovide inputs for selection and/or determination of various featuressuch as poses of instruments, selected targets or identified features(e.g. tumors, anatomical structures, etc.). The user 80 may thereforeview directly the subject 34 and/or additional information relative tothe subject 34 that may be acquired or determined separate from themicroscope or microscope system 30.

As discussed above, according to various embodiments, an image may beviewed with the microscope 32 through the eyepieces 44. The image viewedthrough the microscope may include a live or real view of the subject,also referred to as a real time view. The live view may be produced bydirect visualization of the subject 34. As discussed above the directvisualization may occur due to reflected light or direct light beingtransmitted through the optics of the microscope 32, such as through theobjective lens 52, through internal optics, and finally through theeyepieces 44. Thus, the live view viewed by the user 80 may be of thesubject 34 in real time. Further, as discussed above, various additionalor augmented or mixed view information may be displayed superimposed onthe live view and/or near or adjacent to the live view. Accordingly, theview of the user 80 though the eyepieces 44 may be a mixed viewincluding both the live view of the subject 34 and/or additionalinformation displayed relative thereto. In addition, as discussed above,various views may be displayed to be superimposed on the live view tosubstantially obscure the live view, such as three-dimensionalrepresentation of the subject 34.

Further, as discussed above, the microscope 32, included with themicroscope system 30, may be a robotic microscope that may be moved orcontrolled with the various motors, such as the motors 74, 76 andassociated with (e.g. housed in) the respective arms or members 62, 64.Thus, the microscope system 30 may be positioned near the subject 34 andmoved in a substantially precise and known manner relative to thesubject 34. Alternatively and/or in addition thereto, the microscope 32may be tracked relative to the subject 34 with a selected trackingsystem, including the localizer 114. Further, as discussed above, thetracking system may track the subject and/or the instrument 110 inaddition and/or alternatively to the range finding system including therange finding portions 510, 520.

Accordingly, the position of the subject 34 may be known relative to themicroscope 32, including a view plane of the microscope 32. Thus, thevarious information may be displayed for viewing by the user 80 throughthe eyepieces 44 to represent poses, such as depths, orientations, andthe like, of instruments and/or portions relative to the subject 34(e.g. a target including a tumor) due to the known position of themicroscope 32 relative to the subject 34 and/or the instrument 110.

With reference to the above figures, and additional reference to FIG. 11, a method or process 600 may be used to identify or determine theposition of the additional information relative to the subject 34 fordisplay to be viewed by the user 80 with the eyepieces 44. Additionalinformation may be prior acquired image data or other data, such astractography. Generally, the additional data would be information notviewable in the live view with an optical microscope.

The process 600 may begin in start block 610. After starting the processin block 610, various steps or procedures may occur to determine theposition of the subject 34 relative to other acquired information. Forexample, a determination of a pose of the subject relative to themicroscope eyepiece and/or view plane in block 614 is determined. Asdiscussed above, the pose of the microscope 32 and/or the eyepiece 44and/or a view plane of the microscope 32 may be determined. The pose ofthe microscope 32 relative to the subject 34 may be determined bydetermining an initial position of the microscope relative to thesubject 34 and/or tracking the microscope relative to the subject 34.Various tracking systems, as discussed above, can be used to track ordetermine a pose of the microscope 32 at an appropriate time. Forexample, at a first time or time T1 a pose of the microscope 32 may bedetermined relative to the subject 34 such as with a robotic system,including a motor 74, 76 and related sensors or encoders and/or thetracking device 118. A later movement of the microscope 32 may also betracked using the similar systems to determine a second or later pose ofthe microscope 32 relative to the subject 34.

At an appropriate time, and not required to be after the determination,of the pose of the subject relative to the microscope in block 614,additional information may be acquired in block 618 and a registrationof prior acquired information may be made relative to the subject inblock 620. The additional information may be any appropriateinformation, such as prior acquired image data or other appropriatedata. The additional information may be stored for recall from aselected memory system, such as the memory 90. The additionalinformation may also, however, be generated substantially in real timesuch as illustrating or identifying functional locations on the brain.

The additional information may include information acquired with animaging system prior to the subject being moved relative to themicroscope 32, an image acquired with the microscope 32 (e.g. anoptically acquired image), tractography data, or the like. As discussedabove, the pose of the subject 34 may be determined, such as by trackingthe subject 34 with the range finding system and/or the tracking device120. Further, registration may be made between acquired images and alive view or view with the microscope 32 according to generally knowntechniques, such as identify landmarks in the subject 34 and identifyingsimilar landmarks or identical landmarks in the prior acquired image.

As discussed above, images acquired with the microscope 32 may includemarkings therein that may be identified and may later be compared to thesame positions on the subject 34 for a determination of possiblemovement of the subject 34 relative to the initial pose of the subject34. Similarly and/or alternatively prior acquired images may beregistered to the subject 34 for illustrating appropriate portionsidentified in prior acquired image data (and/or appropriate other data)relative to the subject 34 for viewing with the microscope 32. Thus, theregistration of acquired information may be made to the subject 34 atany appropriate time, such as after tracking or registering the subject34 relative to the microscope 32 and/or registering the subject 34 tothe prior acquired image data in the navigation space which may includemovement and/or volume for moving of the instrument 110 and/or themicroscope 32. Regardless, the registration of the prior acquiredinformation in block 620 may allow for a determination of an appropriatepose of the information in the prior acquired information to the subject34 during a procedure.

The registered additional information may then be determined whether tobe displayed relative to the live view of the subject, such as throughthe eyepieces 44, in block 624. As discussed above the microscope system20 may include various inputs that allow for selections by the user 80.Accordingly, the user 80 may use the inputs 86 to determine whetheracquired information should be displayed. If a determination that noinformation should be displayed a NO block or path 628 may be followedto the start block 610. Thus, the determination of a pose in block 614and registering of information in block 620 may not be required to bedisplayed for viewing by the user 80.

If, however, the determination is made to display the additionalinformation, a YES path 632 may be followed. After following the YESpath 632, a display or illustration of the additional information at theappropriate pose relative to the view plane may be made in block 640. Asdiscussed above, the view plane may be viewed by the user 80 of thesubject 34. Accordingly, the additional information may be displayedrelative to the view plane for appropriate representation to the user80.

For example, as discussed above, a tumor or target may be identified ina prior acquired information that may be substantially below a surfaceof the exterior portion of the subject 34, such as an external surfaceof the brain. Accordingly, a presentation, such as a graphicalrepresentation, of the tumor may be displayed relative to the view planeat the appropriate pose (including depth) for representing theappropriate pose of the tumor for viewing by the user 80 relative to theview pane through the eyepieces 44. The determination of the appropriatepose may be based upon the registration of the prior acquiredinformation to the subject 34 and/or the determined pose of the subjectrelative to the microscope eyepiece or view plane in block 614.Determining the pose of the plane relative to the subject 34 may allowfor an appropriate illustration of the prior acquired information at theappropriate pose relative to the subject 34.

Following displaying 640, the process 600 may end in block end block650. Ending the process in block 650 may include termination of aprocedure, moving the microscope 32 (including, if selected, thesurgical microscope system 30), determining an additional information todisplayed, or restarting the process 600. Accordingly, the end block 650may be understood to be an ending of a particular portion of aprocedure, such as the displaying of a selected prior acquiredinformation according to the process 600.

The microscope system 30 may be used to display appropriate informationfor viewing by the user 80. The information may be displayed based upona determination of a pose of the microscope 32 relative to the subject34 and a registered or known pose of the prior acquired informationrelative to the subject 34. This allows the prior acquired informationto provide appropriate information in perspective to the user 80 due todisplaying the appropriate pose of the prior acquired information to thesubject 34.

The information may be displayed as a graphical representation ofvarious portions, such as an instrument, a target (e.g. tumor oranatomical portion), identified feature or portion, etc. The graphicalrepresentation may be overlaid on a live image, as discussed above. Thegraphical representation, according to various embodiments may be fullyopaque, partially opaque, or transparent. A fully opaque graphicalrepresentation may completely obscure the live image or any portion overwhich it is overlaid. A partially opaque graphical representation allowsfor viewing, at least partially, the portions of the live image overwhich it is overlaid. A transparent graphical representation mayinclude, for example, only an outline and allow for viewing of the liveimage within the outline.

It should be understood that various aspects disclosed herein may becombined in different combinations than the combinations specificallypresented in the description and accompanying drawings. It should alsobe understood that, depending on the example, certain acts or events ofany of the processes or methods described herein may be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,all described acts or events may not be necessary to carry out thetechniques). In addition, while certain aspects of this disclosure aredescribed as being performed by a single module or unit for purposes ofclarity, it should be understood that the techniques of this disclosuremay be performed by a combination of units or modules associated with,for example, a medical device.

In one or more examples, the described techniques may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored as one or more instructions orcode on a computer-readable medium (e.g. memory module) and executed bya hardware-based processing unit. Computer-readable media may includenon-transitory computer-readable media, which corresponds to a tangiblemedium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory,or any other medium that can be used to store desired program code inthe form of instructions or data structures and that can be accessed bya computer).

Instructions may be executed by one or more processors (e.g. processormodule), such as one or more digital signal processors (DSPs), generalpurpose microprocessors, graphic processing units (GPUs), applicationspecific integrated circuits (ASICs), field programmable logic arrays(FPGAs), or other equivalent integrated or discrete logic circuitry.Accordingly, the term “processor” as used herein may refer to any of theforegoing structure or any other physical structure suitable forimplementation of the described techniques. Also, the techniques couldbe fully implemented in one or more circuits or logic elements.

Example embodiments are provided so that this disclosure will bethorough, and will fully convey the scope to those who are skilled inthe art. Numerous specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of embodiments of the present disclosure. It will beapparent to those skilled in the art that specific details need not beemployed, that example embodiments may be embodied in many differentforms and that neither should be construed to limit the scope of thedisclosure. In some example embodiments, well-known processes,well-known device structures, and well-known technologies are notdescribed in detail.

The foregoing description of the embodiments has been provided forpurposes of illustration and description. It is not intended to beexhaustive or to limit the disclosure. Individual elements or featuresof a particular embodiment are generally not limited to that particularembodiment, but, where applicable, are interchangeable and can be usedin a selected embodiment, even if not specifically shown or described.The same may also be varied in many ways. Such variations are not to beregarded as a departure from the disclosure, and all such modificationsare intended to be included within the scope of the disclosure.

What is claimed is:
 1. A method of generating a display for viewing asubject in a view through an eyepiece of a microscope, comprising:determining a first pose of a viewing plane viewed through the eyepiecerelative to the subject; acquiring a first image of the subject at theviewing plane at the first pose at a first time; storing the firstimage; displaying the first image in the view plane superimposed on alive image of the subject.
 2. The method of claim 1, further comprising:analyzing the first image; analyzing the live image; and determiningwhether a change in position of the live image relative to the firstimage has occurred.
 3. The method of claim 2, further comprisingoutputting the determination, wherein outputting the determined posecomprises: generating a graphical representation illustrating thedetermined change; and displaying the graphical representationsuperimposed on the live image.
 4. The method of claim 3, wherein thegraphical representation is partially opaque to allow viewing of thelive image through the graphical representation.
 5. The method of claim3, wherein the graphical representation includes an arrow having anorigin at a point in the displayed first image and a head at a point inthe live image indicating a direction and extent of movement of thesubject relative to the viewing plane.
 6. The method of claim 1, furthercomprising: tracking a head of the microscope relative to the subjectwith the viewing plane at the first pose.
 7. The method of claim 6,further comprising: comparing the live image at a second time to thefirst image after moving the head from and returning the head to theviewing plane at the first pose; determining whether a change inposition of the live image relative to the first image has occurredbetween the first time and the third time; and outputting thedetermination.
 8. The method of claim 6, wherein tracking the headincludes tracking an head tracking device relative to the subject;wherein the head includes an object lens configured to receive lightfrom the subject.
 9. A method of illustrating movement in a view throughan eyepiece of a viewing system, comprising: identifying a landmark on asubject in a viewing plane of the eyepiece at a first time; identifyingthe landmark on the subject in the viewing plane of the eyepiece at asecond time after the first time; evaluating whether a change hasoccurred in the landmark between the first time and the second time. 10.The method of claim 9, wherein the landmark is on a rigid portion of thesubject.
 11. The method of claim 9, further comprising: outputting adetermination of the evaluation of whether a change in pose has occurredincluding superimposing a graphical representation indicating an amountof movement that has occurred between the first time and the secondtime.
 12. The method of claim 11, wherein superimposing the graphicalrepresentation includes: illustrating with a first graphicalrepresentation of the identified location of the landmark at the firsttime; illustrating with a second graphical representation of theidentified location of the landmark at the second time; wherein thesecond graphical representation is visually distinct from the firstgraphical representation to indicate at least one of a change in pose orgeometry or an amount of change in pose or geometry.
 13. The method ofclaim 9, further comprising: forming the landmark on the subject. 14.The method of claim 9, wherein the landmark is a first landmark, furthercomprising: identifying a second landmark relative to the firstlandmark, wherein the second landmark is on a non-rigid portion of thesubject; determining a first pose of the first landmark relative to thesecond landmark at a first time; determining a second pose of the firstlandmark relative to the second landmark at a second time; evaluatingwhether a change in pose has occurred in the between the first landmarkand the second landmark between the first time and the second time;outputting a determination of the evaluation of whether a change in posehas occurred.
 15. The method of claim 14, wherein identifying the secondlandmark include identifying a function position in a brain of asubject.
 16. A system for determining a pose of a microscope relative toa subject, comprising: a first emitter configured to be connected withthe microscope; a first receiver configured to be connected with themicroscope; a processor system configured to communicate with both thefirst emitter and the first receiver; wherein the first emitter isconfigured to emit a first signal that is reflected from the subject andreceived by the first receiver; wherein the processor system isconfigured to execute instructions to determine a pose of a firstportion of the subject based on the emitter and received first signal.17. The system of claim 16, wherein the signal includes infraredradiation.
 18. The system of claim 16, further comprising: a secondemitter configured to be associated with the microscope; a secondreceived configured to be associated with the microscope; wherein theprocessor system is further configured to communicate with both thesecond emitter and the second receiver; wherein the second emitter isconfigured to emit a second signal that is reflected from the subjectand received by the second receiver; wherein the processor system isconfigured to execute further instructions to determine the pose of asecond portion of the subject based on the emitter and received secondsignal.
 19. The system of claim 18, wherein the first portion of thesubject is spaced apart from the second portion.
 20. The system of claim18, wherein at least one of the first signal of the second signal isoperable to reflect from an instrument; wherein the processor system isconfigured to execute further instructions to determine a pose of theinstrument relative to at least one of the microscope or the subject.